A Fixed-Point of View on Gradient Methods for Big Data

نویسنده

  • Alexander Jung
چکیده

Interpreting gradient methods as fixed-point iterations, we provide a detailed analysis of those methods for minimizing convex objective functions. Due to their conceptual and algorithmic simplicity, gradient methods are widely used in machine learning for massive data sets (big data). In particular, stochastic gradient methods are considered the de-facto standard for training deep neural networks. Studying gradient methods within the realm of fixed-point theory provides us with powerful tools to analyze their convergence properties. In particular, gradient methods using inexact or noisy gradients, such as stochastic gradient descent, can be studied conveniently using well-known results on inexact fixed-point iterations. Moreover, as we demonstrate in this paper, the fixed-point approach allows an elegant derivation of accelerations for basic gradient methods. In particular, we will show how gradient descent can be accelerated by a fixed-point preserving transformation of an operator associated with the objective function.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Evaluating the Mixed Marketing Factors of Tehran Big Fish Market from the Sellers' Viewpoint

Evaluation of the marketing mix of the biggest fish market in Iran from the point of view of the sellers was done by questionnaire in 2019. For data analysis, descriptive statistics and Friedman test were used. The results showed that, from the point of view of the sellers, customers prefer to buy consistently from a particular stand or place and their highest priority is salmon and Persian Gul...

متن کامل

Approximate a quadratic mapping in multi-Banach spaces, a fixed point approach

Using the fixed point method, we prove the generalized Hyers-Ulam-Rassias stability of the following functional equation in multi-Banach spaces:begin{equation} sum_{ j = 1}^{n}fBig(-2 x_{j} + sum_{ i = 1, ineq j}^{n} x_{i}Big) =(n-6) fBig(sum_{ i = 1}^{n} x_{i}Big) + 9 sum_{ i = 1}^{n}f(x_{i}).end{equation}

متن کامل

اولویت‌بندی برنامه‌های بهبود سازمان با استفاده از مدل غربالگری فازی

We can call “EFQM excellence model” as a transition from the great number of different models and approaches to unity in context of organizational improvement and excellence.Doing self assessment to find areas for improvement (AFIs) is the most important applications of this model. Organizations specially those ones which are on lower levels of total quality management will encounter lots of AF...

متن کامل

Uncountably many bounded positive solutions for a second order nonlinear neutral delay partial difference equation

In this paper we consider the second order nonlinear neutral delay partial difference equation $Delta_nDelta_mbig(x_{m,n}+a_{m,n}x_{m-k,n-l}big)+ fbig(m,n,x_{m-tau,n-sigma}big)=b_{m,n}, mgeq m_{0},, ngeq n_{0}.$Under suitable conditions, by making use of the Banach fixed point theorem, we show the existence of uncountably many bounded positive solutions for the above partial difference equation...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2017